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NETWORK PROXY APPARATU S AND METHODS 
Field of the Invention 

5 This invention relates to the recordal and use of information about replicas of content 
items in an Internet protocol-based network. 

Background of the Invention 

10 Content items such as HTML files or data files are stored in an Internet protocol jlP) • . _ j , ■ 
based network on servers. The content items may be addressed using, a sc-ca«ed : ^ 
unrversa. resource iocator (URL) of the form la«** ; . ; ,.| 
P rotocol]://lserver.domaInV[directoryVlfllel. Thus for example a URL of the form; . ••' 
http //a.yahc>o.com/some C ontent/skJps.htrnl points to a resource which ,s the file 
15 skips.html located in the -somecontenf directory of server -'a" in the domain 
yahoo.com. The access protocol to be used to obtain the content item is hypertext 
transfer protocol (HTTP). 

It will be seen therefore that the URL mechanism is a combined jiddress both for the. 

20 server location and the location of the file on that server. Additionally the URL identifies 
the access protocol. It has been found that forcing these Information items into one 
single identifier such as a URL is not always an optimum solution. Therefore there are 
alternative mechanisms being considered such as the "handle' mechanism. The 
techniques described below operate equally well with traditional URL based accesses { 

25 and handle-type mechanisms which separate content identification and content;, , 
addressing/location. 

It is the nature of IP based networks such as the Internet Intranets or Extranets that 
content items are cached at various locations on the network. The caching may for 

- -30" - exam^ 

the caching is to reduce the number of file accesses to the original location of the file 
thereby to reduce traffic on the network and increase access s peeds. However, a 
locally cached version of a content item, is often available only to a small group of 
users because the existence of the local replica is not widely advertised.. Thus 
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slough mere may be many copies o, the same content Item J ; 

network, these copies or replicas are available in each case on* to a ema.l M?£ 

In. Thus it may b. ma, a parser user ^ an <-*-•-»» 

.nany thousands of miles away even though there Is an Identic., copy of the content ^ 

SI nearby. There is press* no — » — — • "* 

copy to be accessed. • • ^ 

Several attempts have been made to Improve access to content udng «^^|| 

possibiKy of locally each,*, sems has been discuss* above. ■* 1 ^~gpf 

lnteroach9 protocol (.CP, nave been made and implemented in 

caches to communicate informal about the content they are hottn* Hewe«f 

thl, Is on* pracscabie on a localised basis amends, a sma» numb.- - « ~ 

be used tor exampto on a parte* s«e or wUhln a par*"* £ 
scaleable across a large net-o* such as 1. ~ J"""* ^ 
protocol (defined k, .nteme. Eng*eenng TaskforeenEm RFC 2756) a«c«s 
Lmuntoation batmen caches but is no. sca,eab.e across a large n*wo* such as 



the Internet. 
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Anotnsr a,tem.«ve is ma, offered by Cenus technologies in which content . advised , 
indiscriminately across the Internet Whilst this aUows o*.rs to *,rn about a«en«*W ; 
tocauons for a parficutor content Item. » is no, sceleabie because *ere . 
tocus tor the advertelng funcson and «wetore to be efferfve every user 
network must **• me Cenus advertisement to And ou, about altemasve content. 



locations. 
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Other techniques have been proposed which operate at an original server (I.e. ase^er 
containing the original version of a particular content item). For example, the so<a«ed 
Akamai global load balancing technique provides typically two IP addresses ,n 
response to a request for content for a given domain. The toad balandng technique 
can only operate within that domain and simply chooses the best server to provide me 
particular content item to the requester. The two IP addresses aim to provide a cho,ce 



of the best two servers. 



Summary of the Invention • V 

In accordance with a first aspect, the invention provides a method of indexing cor^n.:| 
an IP-based network comprising intercepting traffic flowing In the network, extract*^ , „ 

5 content Identity information and associated destination location information 

traffic flow, generating a mapping from a content item identified bytre extracted kferfthy g 
information to at least one destination location identified by the associated destine^ |i 
.ocation information, and storing the mapping in a content index database 
operable to provide an instance mapping containing a list of destination 

10 response to an instance request containing a content identity.. 
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This method is described In detail below and is implemented on a content proxy which ; 
na S been termed a "content discovery proxy" (CDP). A CDP may be located near* 
content source in which case it operates to determine where copies or replicas of ; 
particular content Kerns are being sent end then indexes these as alternative locations ; 
for the content items. Alternatively, a CDP may be located near a -ache in which case 
It operates to determine whether particular content items are ston* in the cache and 
then advertises the location back to a content index database (which is termed an 
authoritative proxy in the description below). 

Thus retrieval of content typically will involve searching for an authoritative proxy and, 
interrogating the proxy to see If there are alternative (better) locations for the requested 

content item. * 

This method is typically implemented by a proxy which is termed herein a "content 
request proxy". Therefore in accordance with a second aspect of ±e Invention there J 
provided a method of retrieving content in an IP-based network comprising the steps of 
intercepting a content request containing information related to th* identity of a content 
Item and a source specified location for the content item, sendin £ an instance request 
to a content index associated with the specified source location, the instance request 
including the Identity of the requested content, receiving an Instar ce mapping from the 
content index which contains a list of instances and associated locations for the 
requested content, selecting the best instance of the content from the list, obtaining the 
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. requested content from the location associated with the best instance of the requested 
content, and returning the requested content to the requester of the content. ; , 

. ., ■ ^ahti- 

In accordance with a third aspect, the invention provides a prox/ (termed a 
5 herein) for an IP-based network comprising a data input operable to receive data, 
the network/a data output operable to send data to the network, a* Identity ejdi 
operable to analyse date received at the data input and to ^qToqj^Ugf^ 
information from the data, a location extractor operable to analyse data received at mejy 
data input and to extract location information from the data, a mapping generato^; 
10 operable to generate a mapping from a content Item identified by dentity inforrnation 
provided by the identity extractor, to at least one destination location Identified by 
associated destination location information provided by the location extractor, and* 
content index database operable to store a mapping providec by the mapping 
generator and which is operable to provide an instance mapping containing a list of 
15 destination locations in response to an instance request containing S » content identity. 

in accordance with a fourth aspect the invention provides a pro<y (termed a CRP 
herein) for an IP-based network comprising a data input operable to receive data from 
the network, a data output operable to send data to the network, a location requester 

20 operable to identify a request for a content item in data received at t ie data input end to 
send an instance request to a content index associated with the ecurce location of the 
content item specified in the content request, the instance request Including the identity 
of the requested content, and a content returner operable to receive an instance 
mapping containing a list of instances and associated locations for the requested 

25 content, to select the best Instance of the content from the list, to obtain the requested 
content from the location associated with the best instance of the requested content, 
and to return the requested content to the requester of the content 

In other aspects, the invention provides an advertising message fo:- advertising content 
30 in a cache which has been discovered by a content discovery proxy associated with the 
cache, to a content discovery proxy or an authoritative proxy associated with an origin 

server. 
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,n a further aspect, the invention provides an instance request for transmission over ^ 
, P . baS ed network, the request including a destination address and the identity, ^ 
requested content item, the destination address being a different address to^, 
address of the location of the content item and in another aspect, the Invention prov^r 
5 an instance mapping for transmission over an IP-based network, the Instance map^ng. y 
containing a list of instances and associated locations for a predetermined content Kern.; ^ 

Preferably the step of intercepting traffic is carried out by intercept*,, traffic ftow^ igl 
a cache or alternatively by intercepting content requests from a cache. The ; rneth^ 
10 may further comprise advertising content identity storage mappings whichare stores 
the content Index by sending advertising messages to a predetermined location In the, 

network. 

Correspondingly therefore the invention may also include receiving an adverts** 
15 message having information for creating a mapping or a mapping Itself between a 
content replica and its location, and storing the mapping in a conten : index. 
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The invention 



ion also provides computer programs for operating the atove methods. 
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Embodiments of the Invention will now be described by way of example with reference 
to the drawings. 

Brief descriptift" Q* the drawings 



25 Figure 
invention; 



1 is a schematic block diagram of an IP-based nelwo* Incorporating,^: 



Figure 2 is an exploded view of the portion of Figure 1 shown in doited outline; 

30 Figure 3 is a schematic block diagram of a content discovery proxy in accc^n^ with 
the invention; 1 

Figure 4 is a block diagram of a content request proxy in accordance with the invention, 
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Figure 5 is a schematic block diagram of an authoritative proxy; j^^fe. 

Rgure 6 is a flow chart showing the steps carried out by a CRP; and ' ^ 
5 Figure 7 is a flow chart showing the steps carried out by a CDP. 

nailed description of the embodiments " ( j j H; 
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With reference to Figure 1, a network 2 has logical functions 4-1. 4-il. ™-^^]M^ 
and 4-9. These functions are typicaHy associated «»*>M» 
internet P rotoco.(lP)-based — The network may for example be 
internet, an Intranet or an Extranet The functions send and ^^^^r ; 
network and may for example have data directed to them using an OS, Uyer ^ 
C a web switch") which directs particular types of packets to particular desbnabons. : ; 

The network has two content sources 6-1 and 6-2 which are each 

particular domain. Each of these domains also has an authoritative proxy 8-1 and 8-2 

which is described in detail below. 

The network has client workstation, 10-1. 10-2, 10-3. 1<M end 10* which am; 
operable to request content from different locations on the network 

For example, client 10-1 -nay request an HTML file from domain A Content sou*. 6* 
It does this by resolving a UW. of the fOrm given above into an IP addre» <***^ 
the domain part of the URL to a domain name server (DNSLhaving en .P 
that domain name returned by the DNS and then Issuing a reou.* for content *om , 
particular directory on the server referenced by the returned IP address, ustng a, 
particular protocol such as HTTP. 

, Such a request may, for example, be routed through functions 4-1 and 

arrives at the content source 6-1. The request from the client KM also includes the 
clients own return IP address. Therefore, in the conventional model, the content source 
6-1 extracts the content from its file store and returns it via the network 2 to the 
destination address contained in the original request. The outward and return routes 



between the client 10-1 and the content source 6-1 are not pred.rterrnined and a^ij. ^ 
quite likely to be different from one another as a result of hot potato, "^rf 1 ^^'..^! 
loadings and possibly failures within the network between the times the ortwa^a^,;, '0 
return journeys. 'i ^ 'c. 

It Is possible that the network 2 will have associated with it a cache 2 which Itf 
the requesting client 10-1. The content returned from domain A .x^.sour^MjgM 
may therefore be stored In the cache 12 so that any further request* for, the epr^*» *■> 
the client 10-1 will allow the locally cached copy to be returned ralrWr ; i«ani^j 
10 and returned content needing to flow through substantial parts of the network In: ortffei 
subsequently to obtain the same content. 

With reference additionally to the enlarged portion of Figure 1 she wn in Figure 2, an 
alternative and improved scenario is described which is in accordance with !he 
15 invention. We assume thatthe workstation 10-3 has previously requested the content 
item which has been requested from domain A by workstation 10-1. This item is 
therefore (for the sake of this explanation) assumed to have been stored in a cache 14 
located in the network close to workstation client 10-3. 

20 It will be seen that all traffic passing to and from content source 6-1 is caused to: pass 

through function 4-2. This function (a content discovery |»roxy) has special ; ; 

characteristics. The content discovery proxy 4-2 operates to intercept traffic flowing out; 

of content source 6-1. With reference also to Figure 3. the intercepted traffic passes, . 

into a data input 20 and is passed transparently through to the rej;t of the network yia | 
25 data output 22. Copies of the data are also passed into an identity extractor 24 and a i 

location extractor 26. 

i. 

The traffic is analysed to determine which contact item is being referenced within the 
content source 6-1 and also to which location the content item is being sent This 
30 information is then sent to an authoritative proxy (domain A proxy 8-1 in this case): 
which then generates a mapping between the content item and the location for thai,; 
item, in this case the location for the item is the cache 14. 
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The mapping is then stored in the authoritative proxy 8-1 . (It will be noted;^^.; 
authoritative proxy may form an integral part of the COP or may be a separate <^*)Jg 



Thus, the authoritative proxy builds up a database of locations of content items whfch 
are replicas of the content items stored on the content source 6-1. . . ^ 

in the example being considered, the content item which has been requested *!*«|! 
10-3 and stored in the cache 14 is recorded in the authoritative proxy ; M- 
location of the cache 14 is mapped against it. 
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The use of the database of mappings will now be describe with particular reference to 
Figure 2. the workstation 10-1 makes a content request which passes -vw «xrt»rit,: 
request proxy (CRP) 4-1. With reference also to Hgure 4, the content request passer, 
into the data input 30 of the CRP 4-1 . The CRP passes the date to « location requester, . 



15 32. 



The location requester first checks to see whether the content request (which contain? £ 
a source location for the content item) has an authoritative proxy associated with rt 
One mechanism may. for example, be to insert a unique identlf er Into the domain , 
20 supplied by the client 10-1 such as "registry-. So that if for example the client 10-1 « 

requesting content from the domain a.com, the location requester may look to see rf 
there is a domain »registry.a.com". It may determine this simply by performing a DNS, 

,ooku P on the registry.a.com domain. If no authoritative proxy is available for the, 

location of the content item, the data is passed transparently to the data output 34 an* 
25 backwards compatibility is maintained. However, if an authoritative proxy is avaflabfe,- 

for the domain identified in the request, the location requester *en.is a content request; 

to the authoritative proxy. In this example, the content item is bang requested from 

domain A content source 6-1 and there is an associated authoritative proxy; namely, 

domain A authoritative proxy 8-1 . 

With reference also to Figure 5. it will be seen that the authoritative proxy 8-1 contains, 
for example, two content item mappings (item 1 and Item 2) in its database and has the 
locations of five instances for item 1 and two instances for item 2. The database 
typically contains a URL for each replica and a location in the font of an IP address for 



30 



the server (e.g. a cache server) holding the replica content instance referenced tyftp^ 
URL The authoritative proxy on receipt of the content request irrteriogatesp9|j-:^J| ! 
database to see if the item requested is in its mapping list, if lb. itretums «^|W : g:r ' ' 
URLs and server locations of all the instances of which it is aware. The m***^^ 
5 proxy may also store a tlm*to-ttve for each instance. This is the Known ; pr ^^Mm 
time-to-live of the instance in its location <e.g. the time-to4ive in a cache fr^m|lg^ 
instance) and may be passed back to the CRP for use with the 

feature described below, or may be used to dear expired entries ^^.• a f^W»|^ 
proxy. The information about replica instances is returned ta thft.l«^'-CIW;f^|^||| 

10 instance lisL - 

A content returner 36 then determines which is the best instance. This determination 
is for example, based on the iocation of the servers on which the instances altered ; ; 
or on the performance of the network between these servers and the local node. Since 
15 the CRP 4-1 is close to the workstation client 10-1. it may be assumed that any network ; 
performance measurements made to find the best server which M . made between the 
CRP 4-1 and the location of the instances will accurately reflect the performance 
between the instances and the nearby workstation client 10-1. 
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Having determined the best instance, the content returner 36 recuests the item from, 
the location of that Instance and returns it via the data output 34 to the wortartatkMij 
client 10-1 in the form of a response to the original request.. This occurs transparently; 
so that the client 10-1 has no knowledge of the transactions which have occurreoVaru^ 
the actual source (cache 14) from which the content has been obtained.. Thus in ^ : 
example, it may be seen that the request for the content which hes been replicated or* 
cache 14 is effectively redirected so that instead of obtaining the content item from the, 
content source 6-1. workstation client 10-1 received the content Instead from cache 14 
which is chosen by CRP 4-1 to be a better location from which to obtain the item. 

It will be noted that generally speaking a CDP will be located near a content source or a 
caching location for content and that a CRP will generally be located near the part of 
the network in which requests for content are generated. However. In some situations, 
these two locations are the same and thus, for example, function 4-9 may operate as a 
combined CDP and CRP. 
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As another example of CDP operation , . CDP 4-9 may monitor the contents of a 
40 and advertise the items which are held there, to appropriate au ^^ : ^ 
Thus, assuming that the cache 40 holds an item which originate from 
content source 6-2. the proxy 4-9 may detect this item entering the cache^, 
advertise the location to the domain B authoritative proxy for 8-:>. 
because » Knows the source ioca^on of the item because . ^ 
address and URL the item was obtained. It may then determme v^emeri^| 
associated authoritative proxy w«h the domain represented ^^W^ 
, and then advertise the content back to the authoritative ^ ^ ^"^^^ 
..egistr/ mechanism described above, in this case, it is likery that «. CDP 4* 
has made the authoritative proxy aware of the sending of the items to the cache 40, , 
However if the CDP 4-5 were to be omitted the advertising message from th* j 
CDP/CRP 4-9 would be of value. 

5 „ w.1. be noted that the cache. 14 end 40 may communicate with one another udno 
.CP for example, end transfer cached content bet*™ th«ns«w». This may be. 
monitor* by me CDP* 4-7 and/or 4« The duplicate or transfer of cached data n«y 
then be adventeed back to a relevant authortta*ve proxy. 

" As a further enhancement, the Instance lists mey theme. M. b* cacr-d. T*u*«i ; 
opto*, proxy reals** 42 may be to «» '"stance Us* wh*h have 

obtained. tla-mwi^.-d-^^^^^^TSfti 

content cache. Thus if the wo*eta«on cHent 10-1 were to ^ ^ 
2S instance lis. had a,ready been mceived. the CRP may detem-ne this * <M-V <"» 
optional proxyregistry 42 andusing the locany cached Instance fel: instead. 

Thus mis arrangement allows a content provider to allow fast access to its -MM 
local sources of the content. It also allows the content provtdsr to be aware of the 

30 location of replicas of the content. 

It will be appreciated that any protocol may be used for the communications between 
the CRPS and CDPs and the authoritative proxy. However, it is fortuitous that the 
HTCP proposal has suitabie fields for these communications even though rt was not 
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originally designed for this purpose, 
communications. 



10 



15 



Thus HTCP is a suitable protocol .for isu|hi 
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in a preferred embodiment, a CDP and/or CRP would be formed as s « lnte^ t|^||i|||| 
local cache. For example the CRP 4-1 would be integrated with the .c^;l2*^pg 
combined CDP/CRP 4-9 would be integrated with the cache 40. Th* ^^f|||f ^ 
to know exactly what content Is available on the cache at any one time.: 

Alternatively, a CDP may be provided for implementation with a teg^ycache. 

Typically, there are four types of caching in use. The simplest type of caching is a so- 
called -inline cache" such as the cache 4* in Figure 1. This as its name suggests 
inserted inline and typically operates transparently by receiving requests for content, , 
checking whether it already holds the content and if not. making a request forlhe 
content from the content source. The cache caches content according to its own 
interna, policies and passes the content (obtained either from the: source or its own . 
cached storage) transparently through to the next function or node in the network, 

Alternatively, the cache may not be 'inline- but may be used with * rnechanism which V 
directs content requests and data into the cache. For example, a client may be! ; 
configured with a web proxy address which Is actually the address of the cache. .Thus,, 
any content request made by the client will always be directed to the cache an* th*:7 
cache may then meet the request from its own stored content or request the content 
from the source location and pass it back to the client Another example {discussed; 
briefly above) Is an OSI layer 4 switch which looks for particular types of packets and 
directs them to a cache server. A further option is to associate a cache with a -web 
cache control protocol" (WCCP^enabled router. WCCP allows a router to 
communicate with a cache in a way that allows the router and the cache to co-operate 
to cache traffic flowing through the router according to predetermined policies. 

in the case of the latter three examples of caches which are non-inline, the CDP will- 
typically be located near the cache in the network. The mechanisms described above; 
either monitor traffic flowing into the cache or communicate directly with the each* 
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using a protocol such as ICP or HTCP. which then allows i^<^.i^^^^ 
content. ,,• . ^ 'j^Hjf||||| 

in the case of the inline cache, it may be possible to co-locate a CDP vrfththe 
5 and to communicate using a protocol such as ICP or HTCP. 

An alternative approach for Inline caches is to use a CDP located at Ihe source tocaffcn, j 
which looks for content being directed to a cache. It may. for example, recpgrts^ 
content going to a cache because the HTTP protocol contains partkwlar text 8^ ^ ; 
10 Thus a CDP located at the source location can inform the local authoritative proxy*** : ; 
located at the source location that content has just been passed to what appears; to be 
a cache. 

As a further enhancement, any of the functions above such as an arthorttatlve proxy. a (■. 
16 CDP or a CRP may pro-actively create replicas on the network and record them in the, $ 
authoritative proxy. For example, it may be determined that particular fil* or j: 
directories are likely to be used heavily at particular locations. Thus a cache local to,, 
those locations may be provisioned in advance with replica Instances. 

20 The CDP ideally should determine the lifetime of items stored on the cache in order to 
avoid directing CRPs to the cache only to have the CRPs find ttwt the item does not 
exist on the cache any more. A CDP may therefore monitor how frequently the cache 
requests a particular item in order to determine how quickly the items are expired or it . 
may for example be able to "talk" to the cache for example using H"CP. The CDP may 

25 also use the expiry time contained in the HTTP standard (or equivalents) as a backstop 
expiry time for content. The CDP may also keep statistics about failed request* to 
retrieve items from the cache and to advertise the absence of a pa rticular content item 
in its cache to the relevant authoritative proxy. 

30 In another preferred embodiment, the CRP may form part of a web proxy or may fbrj 
example be the recipient of particular protocols (for example HTTP requests) as a; 
result of a web switch which directs traffic for particular transport protocols to the CfP. ; 
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It will also be noted that if the CRP is located away from a particuUir ,wo^sto^(gi 
readily scaleable to several local users. Furthermore, the CRP' meiy. p^rfpr^^***'^'' 
role. It may, for example, be used to hide the actual locations content ttenjsj-1 
client. The client may make a request which is effectively tranela^ by the ^ 
conjunction with the authoritative-proxy, into a real address-for a particular.ca^j 
This may be used to prevent direct access to high security items without first being.:: if 
authorised by the CRP. 




With reference to Figure 6, the steps undertaken by the CRP are In general tei^a£f 
10 follows. 

The CRP receives a content request from a client (step 50). The CIV then performs a 
DNS look up (step 52) to determine if an authoritative proxy is available for the domain ; ; 
associated with the content In the content request. 

15 ' . ' ih^-'v 

If an authoritative proxy is not available the content is obtained from the ongin spaefflBd^ 
in the content request (step 54).the item is returned to the requester (step 56) arid r#| 
be cached (step 57) \ } } 

20 If an authoritative proxy is available (step 58), an instance request is sent to the 
authoritative proxy (step 60). The CRP then receives an instance list (step 62), which 
may be cached (step 63), selects the best instance (step 64) obtains the best instance 
of the item from the located associated with that instance (whicn is returned In the 
instance list) and then returns the item (step 56). The item may also be cached (step 

25 57). 

With reference to Figure 7, the steps carried out by a content discovery proxy are set;; 
out in broad terms. 

30 In step 70, the CDP intercepts network, for example, traffic flowing into inline cache, drj 
interrogates a cache, for example, using HTCP, 

In step 72, in the case of intercepted traffic the CDP extracts content Mepttty 
information and location information from the traffic. In the case of communications 
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direct,, with a cache, I. tocaton of the cache h known and the content identity -sjf^..^ 



extracted from the cache. 
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The CDP men checks to see whether there is an authoritative proxy 

forming the source *c*i=n o, the «*- (^^•^l^^ 
» found the CDP passes the content identity and <^T^M^0 
author^ proxy *» ^ * ~ 

Information is discarded (step 80, and the ioop is resterted tr^ft^gMgl 

additional content replicas. .. ! > liT^'.^i^ 

The invent*, described above described above provides a raadily aca**^ '■'^ 
icemen, for improving access .—I " 
witt, content servers which do not have authoritative proxies and nay be 
legacy- hardware pattern,*. A typical ro» out sc««no might be the 
CDPs and authoritative proxies at selected hJgh traffic content sourcas and the, 
provision of ORPs by users wteh.no to take advantage of eu*ontetve I""*-***- ; 
providing CDP functionality within the CRP, the mappings contained in the authoritative 
proxy may be made more comprehensive. 



